3-Dimensional image processing method, 3-dimensional image processing device, and 3-dimensional image processing system

ABSTRACT

Illumination light having a 2-dimensional luminance distribution is used for obtaining photographed image data, and surface attributes of a real object are estimated based on photography environment information such as information relating to the luminance distribution of the illumination light at the time of photography, along with the photographed image data. This solves a problem with 3-dimensional image processing devices which obtain information relating to surface attributes of a real object from photographed image data obtained by photographing the real object along with the 3-dimensional shape of the real object and reproduce the object as a 3-dimensional image on a display screen, wherein specular reflection may not be captured in the photographed image data with objects having metal glossiness or the like where the specular reflection only occurs at an extremely narrow angle range at the time of obtaining the surface properties, resulting in erroneous information with no specular reflection being input. Thus, photographed image data of the real object is obtained without failing to pick up any specular reflection light.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates to a 3-dimensional image processingmethod, 3-dimensional image processing device, and 3-dimensional imageprocessing system, and particularly relates to 3-dimensional imageprocessing method, 3-dimensional image processing device, and3-dimensional image processing system, wherein information of specularreflection is obtained without fail even when photographing objects withextremely narrow angle ranges of specular reflection.

[0003] 2. Description of the Related Art

[0004] In the event of reproducing a real object as a 3-dimensionalimage using computer graphics, there is a technique used wherein imagestaken of the real object with a camera are applied to each portion ofthe real object being 3-dimensionally reproduced. With such a technique,multiple images taken from multiple directions are prepared so that the3-dimensional image can be observed from various viewpoints, and imagessuitable for the viewpoints are selected at the time of reproduction,and applied to each portion of the shape of the 3-dimensionallyreproduced real object.

[0005] However, there is a problem with this technique, in thatreproduction can be made only regarding the illumination state used atthe time of inputting data from the real object, and reproduction cannotbe made under an illumination state which is different to that at thetime of photography. Also, camera images already contain shadows andhighlights, so further applying shadows and highlights to objects towhich camera images have been applied results in an unnatural-appearingimage.

[0006] Now, SIGRAPH Computer Graphics Proceedings, Annual ConferenceSeriec, 1997, pp. 379-387, “Object Shape and Reflectance Modeling fromObservation” describes the following technique. At the time of inputtingdata from a real object, the surface attributes, which are change in themanner of reflection observed on the surface of each of the parts of thereal object due to the direction of illumination and the direction ofobservation (photography), are represented as a BRDF (Bi-directionalReflectance Distribution Function) and thus held, so in the event thatan illumination state is arbitrarily set at the time of reproducing the3-dimensional image, a 3-dimensional image can be observed as if thereal object had been placed under the arbitrarily set illuminationstate. Also, a technique has been proposed wherein predeterminedreflection model functions are introduced as BRDF, so as to representthe surface attributes by a reflection constant substituted into thereflection model function.

[0007] With this technique, a predetermined reflection model functionserving as a BRDF is applied to the reflection properties of each partof the real object obtained by photographing the real object, and thereflection properties of each part of the real object are represented bythe reflection constant within the reflection model function, and thusheld. At the time of reproducing the 3-dimensional image, observationconditions such as arbitrary illumination conditions, the line of sight,and so forth, are set, and reflections occurring at each of the parts ofthe real object under those conditions are computed from the heldreflection constant and reflection model function, thereby enablingshadows and highlights which are natural under the set observationconditions to be realized.

[0008] Also, for the above-described reflection model function, areflection model function wherein the reflection of light is taken to bea linear sum of scattered (diffuse) reflection components and specularreflection components and wherein the state of the surface can beexpressed by multiple reflection constants each, is used. The Phongreflection model is a representative example of such.

[0009] However, in the event of reproducing 3-dimensional images byholding data of the real object in such a technique, there is the needto use a light source which can approximate point light sources andparallel (infinite-distance) light sources as the illumination lightsource for inputting data from the real object, due to the properties ofthe BRDF. The reason is that since the BRDF is a function whichdescribes what sort of reflection light the surface of the real objectcreates upon incidence of illumination light at a constant incidentangle, prediction becomes extremely difficult when using image datataken of a real object illuminated with an illumination light of whichlight source position cannot be determined.

[0010] In the event of using such a light source, specular reflectionmay not be observed in the photographed image data with real objects inthe case of real objects with metal glossiness or the like wherein thespecular reflection only occurs at an extremely narrow angle range,resulting in erroneous information with no specular reflection beinginput. In order to avoid this, photography of the image needs to becarried out at angle intervals smaller than the expansion of thespecular reflection, but this in itself is a problem, since this greatlyincreases the amount of image data to be processed and troublesomephotography tasks.

SUMMARY OF THE INVENTION

[0011] Accordingly, it is an object of the present invention to providea 3-dimensional image processing method and device, capable of obtainingphotographed image data of the real image by spreading the regionwherein specular reflection occurs even with real objects whereinspecular reflection is only observed in an extremely narrow angle range,by illuminating the real object with a light source having a luminancedistribution with 2-dimensional expansion as observed from the surfaceof the real object, thereby preventing failure to detect specularreflection.

[0012] Also, in the event that photographed image data is obtained by anillumination light source having a luminance distribution with2-dimensional expansion as observed from the surface of the real object,the data directly obtained from the photographed image data is datacomprising multiple BRDFs overlaid (or a convoluted BRDF). A BRDFindicating reflection properties of the surface of the real objectobtained with regard to illumination light not having 2-dimensionalexpansion is obtained by computation from the aforementioned datacomprising multiple BRDFs overlaid, by inputting the photographenvironment information at the time of photography (information relatingto luminance distribution of the light source, information relating toproperties or settings of the photography device which executes thephotography, information relating to the position and direction of thereal object and light source and photography device, and so forth).

[0013] Thus, failure to observe the specular reflection at estimatedportions (color portions) of the surface attributes in the event thatthe specular reflection is sharp can be prevented, and proper surfaceattributes can be estimated in a stable manner, with a relatively smallnumber of photographed images.

[0014] Also, prediction of the BRDF obtained from the obtained data inthe event of observing with point light source can be facilitated, bysetting a Gaussian distribution-type reflection model function as thereflection model function to apply to the BRDF, and using a light sourcehaving Gaussian distribution-type luminance distribution for the lightsource as well.

[0015] Further objects, features and advantages of the present inventionwill become apparent from the following description of the preferredembodiments with reference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0016]FIG. 1 is a configuration diagram of a 3-dimensional imageprocessing system according to a first embodiment of the presentinvention;

[0017]FIG. 2 is a block diagram illustrating the configuration of a3-dimensional image processing device making up the 3-dimensional imageprocessing system according to the first embodiment;

[0018]FIG. 3 is a diagram illustrating an example of obtaining imagedata of a real object with the 3-dimensional image processing systemaccording to the first embodiment;

[0019]FIG. 4 is a flowchart illustrating processing for estimating colorinformation with the 3-dimensional image processing system according tothe first embodiment;

[0020]FIG. 5 shows expressions for a Phong reflection model function;

[0021]FIG. 6 is a diagram illustrating an example of obtaining imagedata of a real object with the 3-dimensional image processing systemaccording to a second embodiment of the present invention;

[0022]FIG. 7 shows expressions for a Gaussian distribution-typeluminance distribution reflection model function;

[0023]FIG. 8 shows expressions for a Gaussian distribution-typereflection model function;

[0024]FIG. 9 is a flowchart illustrating processing for estimating colorinformation with the 3-dimensional image processing system according tothe second embodiment;

[0025]FIG. 10 shows a determinant wherein the Phong reflection modelfunction has been applied with the first embodiment;

[0026]FIG. 11 shows a determinant wherein the Phong reflection modelfunction has been applied with the first embodiment; and

[0027]FIG. 12 is an explanatory diagram of the parameters θ and ρ usedin the reflection model functions.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] First Embodiment

[0029]FIG. 1 shows a configuration diagram of a 3-dimensional imageprocessing system according to a first embodiment of the presentinvention. With the present embodiment, the luminance distribution ofthe reflected light at the surface of the real object that has beenobtained by illuminating the real object with a light source havingluminance distribution with a 2-dimensional expansion as viewed from thesurface of the real object is deconvoluted using the luminancedistribution of the illumination light, and the bi-directionalreflectance distribution function BRDF is estimated to reproduce a3-dimensional computer graphics. A Phong function is used here as thereflection model function for the BRDF.

[0030] The 3-dimensional image processing system according to thepresent embodiment is configured of the following components. A subject(real object) 3 is placed on a rotating stage 4. The subject 3 isilluminated by a planar area light source 2 having luminancedistribution with a 2-dimensional expansion. A photography device 1 is adigital camera for performing photography with an imaging device such asa CCD, for example, which obtains photographed image data of thesubject. Also, a 3-dimensional measuring device 5 measures the3-dimensional form of the subject 3 using the laser radar method, slitlight projection, pattern projection, etc. A 3-dimensional shape model,which is 3-dimensional shape information of the subject 3 based on theactual measurement data input from the 3-dimensional measuring device 5,is generated by a 3-dimensional image processing device 6. Also, thesurface attributes (coloring information such as color, reflectionproperties, etc.) based on the photographed image data input from thephotography device 1 is estimated at the 3-dimensional image processingdevice 6, and model data for the subject 3 is generated for the subject3 from the surface attributes data and the 3-dimensional shape model. Acomputer graphics creating device 7 creates 3-dimensional computergraphics (a 3-D image) from the model data of the subject 3 created bythe 3-dimensional image processing device 6, and displays the created3-dimensional computer graphics.

[0031] The 3-dimensional shape model generated by the 3-dimensionalimage processing device 6 may be represented as polygonal surface model,for example, or as a collection of surface shape components of differentshapes.

[0032] The 3-dimensional measuring device 5 and the photography device 1are each positioned at locations distanced from the subject 3 by acertain degree, and perform 3-dimensional shape measurement andphotograph of the subject 3 each time the subject 3 on the rotatingstage 4 rotates a certain rotational angle, thereby obtaining andoutputting actual measurement data of the 3-dimensional shape andphotographed image data. Note that while in FIG. 1, the 3-dimensionalmeasuring device 5 and the photography device 1 are separatelyconfigured, these may be configured as an integral unit. Also, the3-dimensional measurements and the photography may be performedseparately or at the same time.

[0033] Next, each of the processes performed by the 3-dimensional imageprocessing system will be described with reference to FIG. 2 along withFIG. 1. FIG. 2 illustrates the configuration within the 3-dimensionalimage processing device 6. The 3-dimensional image processing device 6is configured of an image data obtaining unit 10 which obtainsphotographed image data, a photography environment information obtainingunit 11 which obtains information relating to the position and shape ofthe illumination light at the time of photography, the position of thephotography device, etc., a 3-dimensional shape obtaining unit 12 whichobtains a 3-dimensional shape model from the 3-dimensional measuringdevice 5, a color information estimating unit 13 which estimates thesurface attributes, primarily from the photographed image data and thephotography environment information and the like, and a model datastorage unit 14 which stores the 3-dimensional shape model generated atthe 3-dimensional shape obtaining unit 12.

[0034] As shown in FIG. 1, with the present embodiment, the model datastorage unit 14 is provided as a model data storage device separatelyfrom the 3-dimensional image processing device proper. Also, at thecolor information estimating unit 13, surface attributes may beestimated taking into consideration the 3-dimensional shape modelgenerated at the 3-dimensional shape obtaining unit 12, besides thephotographed image data and the photography environment information, asdescribed separately. Also, in the event that the spectrum of theillumination light from the light source is known, the dispersereflection and specular reflection components may be separated, and thesurface attributes may be estimated from the photographed image data andthe photography environment information, without taking the3-dimensional shape model into consideration.

[0035] (Obtaining Image Data)

[0036] The following is a description of obtaining image data. First,the image data obtaining unit 10 obtains image data from the photographydevice 1, and outputs this to the color information estimating unit 13.

[0037] Now, photography of the subject 3 by the photography device 1will be described with reference to FIG. 3. With the present embodiment,the subject 3 placed on the rotating stage 4 in a darkroom isilluminated with a constant illumination light cast from the area lightsource 2 having luminance distribution with a 2-dimensional expansion.Illuminating from the area light source having luminance distributionwith a 2-dimensional expansion allows a finite and continuousdistribution to be created at the incident angle of the illuminationlight cast on the subject 3, and thus a finite expansion is created inthe direction of the reflected light off of the subject 3 due tospecular reflection. Accordingly, failure to detect specular reflectionon the subject 3 at the time of obtaining image data can be prevented,and further, estimation of surface attributes can be performed moreaccurately since continuous data can be obtained.

[0038] As for the size of the area light source 2, prevention of failureto detect specular reflection light can be effected in the event thatthe maximum solid angle of apparent size, when observing the area lightsource 2 from each part of the subject 3, is around 5° to 10°, orgreater. Also, in the event of obtaining image data from subjects havingparts with particularly small curvature radii, or subjects creatingsharp specular reflection, the above maximum solid angle of apparentsize is preferably 20° or greater. Also, the area light source 2 may belarge enough to encompass the subject 3 if the area light source 2 has adark portion around 5° to 10° or greater as observed from each part ofthe subject 3. Note that the form of the area light source is notrestricted to a 2-dimensional planar form; rather, the area light sourcemay be, for example, rod-shaped, ring-shaped, or any other shape so longas the apparent size viewed from each part of the subject 3 is equal toor greater than the above-described angles. Further, either a singlearea light source or multiple area light sources may be used forobtaining the image data.

[0039] The luminance distribution within the area light source 2 needsnot be uniform over the entire surface thereof, as long as the luminancedistribution is known or can be measured. Also, in the event that theapparent size is great when observing the area light source 2 from eachpart of the subject 3, to the extend of encompassing the subject 3, theluminance distribution is preferably not uniform over the entire solidangle.

[0040] The image data is obtained by rotating the rotating stage 4 whileilluminating the subject 3 with the area light source 2, multiple shotsare taken with the photography device 1 each time the rotating stage 4rotates a certain rotational angle, and image data made up from thethree color bands, R, G, and B, is obtained.

[0041] While carrying out this, the position (placement) of thephotography device 1 and the subject 3 and the area light source 2, theluminance of the area light source 2 (the illumination light), thecamera parameters (exposure time, sensibility, focal length, etc.) ofthe photography device 1, and other such photography environmentinformation is recorded each time a photograph is taken.

[0042] Thus, performing photography multiple times while rotating thesubject 3 changes the relative illumination direction and observationdirection of the subject 3, yielding multiple images (an image group)taken changing the light source direction and line of view.

[0043] Note that while the present embodiment is described withreference to a case wherein the subject 3 is placed on a rotating stage4, the subject may be placed on a device having multiple rotationalaxes. Also, while the present embodiment is described with reference toa case wherein the photography device 1 and the area light source 2 arefixed, with the subject 3 being rotated, arrangements may be madewherein the subject 3 is fixed and the position and/or the angle of thephotography device 1 and/or the area light source 2 is changed, orwherein and the position and/or the angle of two or more of thephotography device 1, the area light source 2, and the subject 3, ischanged. In any case, any arrangement will suffice as long as multipleimages with difference light source directions and/or lines of view canbe obtained by changing or rotating positions.

[0044] (Obtaining photography environment information)

[0045] The photography environment information obtaining unit 11 obtainsphotography environment information for each image photographed, such asinformation relating to the luminance distribution of the area lightsource 2 (illumination light), information relating to the cameraparameters and settings of the photography device 1, informationrelating to the positions and attitudes of the photography device 1, thearea light source 2, and the subject 3, and so forth, and outputs thisto the coloring information estimating unit 13.

[0046] Note that part of the photography environment information may notbe recorded at the time of photography, but rather estimated from theimage data. As an example of a method for estimating photographyenvironment information from the image data, a method may be usedwherein markers fixed on the subject 3 at the time of photography arephotographed along with the subject 3. With this method, the shape ofthe markers and the surface attributes are measured beforehand, and thephotography environment information is estimated by photographing themarkers along with the subject 3. Also, in the event that there arecharacteristic parts on the subject 3 of which shapes and surfaceattributes are known, and capable of substituting for the markers, thephotography environment information may be estimated using these inplace of the markers. The photography environment information thusestimated is output to the coloring information estimating unit 13.

[0047] (Obtaining 3-Dimensional Shape)

[0048] The 3-dimensional shape obtaining unit 12 generates a3-dimensional shape model based on actual measurement data from the3-dimensional measuring device 5, which is output to the coloringinformation estimating unit 13, as well as recording the 3-dimensionalshape model in the model data storage unit 14.

[0049] Note that the 3-dimensional shape model may be obtained using a3-dimensional scanner or the like at the time of image photography, ormay be estimated from image data. Or, in the event that shape data isknown from CAD data or the like, such as with the case of industrialarticles, that data may be used.

[0050] (Estimating Coloring Information)

[0051] The coloring information estimating unit 13 estimates thecoloring information which is the surface attributes information of thesubject, based on the input information, and records reflectionconstants for the disperse reflection and specular reflection making upthe surface attributes in the model data storage unit 14. The followingis a method for estimating the coloring information, with reference tothe flowchart in. FIG. 4.

[0052] First, in step S1, the flow starts. Subsequently, the followingprocessing is repeated for each coloring portion (estimation portion)for which the coloring information of the surface of the 3-dimensionalshape model is to be estimated.

[0053] In step S2, the set of the direction of the center of the lightsource, the direction of observation, and the reflection intensity, iscalculated at each coloring portion and a table of the set is compiled.This is achieved by calculating the direction of the center of the lightsource at each coloring portion, the direction of observation, and theposition on the image, from the positional information of thephotography device 1, subject 3, and area light source 2 at the time ofphotography, and properties or settings information for the photographydevice 1 and area light source 2, and obtaining a BRDF′ (lc, θo) whichis an apparent BRDF which does not take into consideration the fact thatthe light source is an area light source for the lightness on the image.Here, the BRDF is represented as a function having the direction of thecenter of the light source lc and the observation direction θo asparameters thereof. Also, in the following description, values orfunctions containing a 2-dimensional luminance distribution ofillumination light as a component thereof are indicated with a prime(′).

[0054] In the event that the reflection intensity is unknown due to theplane being the back side on the image, behind other members, or outsideof the image, this is not included in the table.

[0055] From step S3 on, tasks are carried out for estimating the realBRDF which is equivalent to a case wherein the incident angle to eachpart of the subject obtained by illumination such as a point lightsource or parallel light is concentrated, from the BRDF′ which is theapparent BRDF obtained from the area light source 2. Specifically, theluminance distribution of the area light source is represented with thefunction l(r) of the relative direction r from the direction of thecenter of the light source at each coloring portion, the luminancedistribution is deconvoluted based on the following expression, andconverted into a bi-directional reflectance distribution function BRDF(θi, θo) yielding the reflectivity of light without 2-dimensionalexpansion, which is a function of illumination direction and observationdirection.

[0056] Expression 1

[0057] Illumination luminance distributionB  R  D  F^(  ′)(θ  i, θ  o) = ∫_(r)l(r) * B  R  D  F  (l  c + r, θ  o)r

[0058] From step S4 on, fitting reflection model function to this BRDFis performed. In the present embodiment, the Phong reflection modelfunction shown as a reflection model function in FIG. 5 is used. Withthe Phong reflection model function, the relation between the directionof center of the light source at each coloring portion, the observationdirection, and reflection intensity, in the table obtained in thepreceding step, is represented with the reflectivity R_(j), G_(j), andB_(j), for each color band, and the angle variables θ_(j), and ρ_(j)shown in FIG. 12.

[0059] Note that in FIG. 12, θ is the angle between the surface normaland the direction to the center of the light source from a coloringportion, and ρ is the angle between the surface normal and half vector,which is an intermediate vector direction between the direction ofobservation (photography) by the photography device 1 the direction toand the center of the light source.

[0060] Now, the subscript j represents image Nos. 1, 2, and so onthrough m. Also, Cd_(R), Cd_(G), and Cd_(B) are diffuse reflectionconstants, with Cs,n being a specular reflection constant. With an errorvector as ε, this yields the determinant shown in FIG. 10. In the eventthat the reflectivity at the coloring portion is unknown, thecorresponding set is deleted from the table.

[0061] Next, in step S5, a set of constants optimal for fitting thetable with a reflection model function, i.e., the set (Cd_(R), Cd_(G),Cd_(B), Cs′, σ′) is decided, and recorded in the model data storage unit14. Various types of mathematical approaches may be used foroptimization; the present invention does not restrict the approach usedhere.

[0062] As an example, in the event of using the least-square method, theperformance function is |ε|, the length of the error vector ε, and a setwherein the performance function is minimum is obtained. Though theperformance function handles all errors as being equivalent, aperformance function may be set which takes into consideration thereliability of each of the values. Particularly, weighting proportionateto cosθi and cosθo is effective. This processing is performed for allcoloring portions.

[0063] A 3-dimensional image is reproduced by combining the3-dimensional shape model, and the data relating to the surfaceattributes of each part of the 3-dimensional shape model represented bythe reflection model function constants, thus obtained.

[0064] (Reproduction by Computer Graphics)

[0065] The following is a description of the steps for generating anddisplaying the subject 3 as 3-dimensional computer graphics, using themodel data. At the computer graphics generating device 7, informationrelating to the observational position and the position of the subjectin a virtual space, and information relating to a virtual light source(position, direction, color, illuminance, etc.), are input. Next, thisinformation is substituted into the 3-dimensional shape model and thereflection model function which is the surface attributes at eachportion of the 3-dimensional shape model obtained above, and how thesubject would be observed under the input observation environment iscalculated. The product of the calculated reflectivity and illuminationilluminance is drawn on the display position on the generated image as adisplay color. Thus, natural shadow and highlights under arbitraryillumination conditions and direction of line of view are realized.

[0066] Note that while the present embodiment has been described withreference to an arrangement wherein the camera 1, area light source 2,rotating stage 4, 3-dimensional measuring device 5, 3-dimensional imageprocessing device 6, and computer graphics generating device 7 areseparate, these may be configured integrally.

[0067] Also, in the event of configuring a 3-dimensional imageprocessing system with multiple devices or pieces of equipment as withthe present embodiment, exchange of data between the devices orequipment is carried out by means such as various types of recordingmedia, wireless or cable data communication means, and so forth.

[0068] Particularly, providing the 3-dimensional image processing device6 and computer graphics generating device 7 as separate devices isadvantageous in that model data once created can be repeatedly used withmultiple computer graphics generating devices.

[0069] Also, with the present embodiment, the reflection model functionhas been described as being a Phong reflection model function, but thepresent invention is by no means restricted to this reflection modelfunction; rather, other reflection model functions may be used instead,such as the Torrance-Sparrow model, for example. That is to say, anysuch reflection model function may be used with the present invention solong as the reflectivity is represented as one or multiple constants andangle variables.

[0070] Also, while the present embodiment has been described withreference to an case wherein the image is made up of the three colors,R, G, and B, the present embodiment may use any arrangement as long ascorresponding reflection model functions can be described, such as onecolor, other multiple colors, spectrum information, polarizationinformation, etc.

[0071] Further, the coloring information estimating method described inthe present embodiment does not guarantee that reflection constants areuniquely obtained for all portions, but in such portions, techniques canbe used such as interpolation of coloring portions having unobtainableconstants based on reflection constants at nearby coloring portions,substituting a typical value, and so forth.

[0072] Second Embodiment

[0073] The above first embodiment has been described with reference to aprocess for deconvoluting the apparent BRDF relating to the real objectwith the luminance distribution of the illumination light in order toestimate reflection constants in the reflection model function, whenapproximating the BRDF with a predetermined reflection model function.Conversely, with the present embodiment, a technique will be describedwherein an area light source having a 2-dimensional luminancedistribution capable of being approximately with Gaussian distribution,and also wherein a Gaussian distribution type reflection model functionis used. Such a configuration simplifies the computation fordeconvoluting the apparent BRDF into the actual BRDF.

[0074] The configuration of the 3-dimensional image processing systemfor performing the processing with the present embodiment is the same asthat described with the first embodiment, so the devices and equipmentmaking up the system are denoted with the same reference numerals.

[0075] The following is a description of the processing steps with the3-dimensional image processing system according to the presentembodiment.

[0076] (Obtaining Image Data, etc.)

[0077]FIG. 6 shows an overview of the configuration for obtaining imagedata with the present embodiment. With the present embodiment, an arealight 22 having a luminance distribution with 2-dimensional expansion,wherein the luminance distribution thereof is approximated Gaussiandistribution with an angle radius σ_(L), is used as the light source forilluminating a subject 3. Such an area light 22 can be configured usingfrosted glass with a spotlight casting light from behind, for example.

[0078] The luminance distribution here is a distribution on a spherewherein illumination is projected from each direction viewing the arealight source 22 from each portion of the subject 3, and since this is inplanar approximation with the region wherein the area light source isprojected on the sphere with the present embodiment, the maximum angletherein should be 5° to 10° or more, but 90° or less.

[0079] Multiple sets of image data relating to the subject 3 areobtained at the image data obtaining unit 10 from the photography device1, using such an area light source 22, and the information is output tothe coloring information estimating unit 13.

[0080] The photography environment information obtaining unit 11 obtainsphotography environment information such as information relating to theluminance distribution of the area light source 22 (illumination light),information relating to the properties and/or settings of thephotography device 1 information relating to the position and/orattitude of the photography device 1, subject 3, and area light source22, and so forth, each time a photograph is taken, and outputs thephotography environment information to the coloring informationestimating unit 13.

[0081] At this time, part of the photography environment information maynot be recorded at the time of photography, but rather estimated fromthe image data, as described with the first embodiment.

[0082] The 3-dimensional shape obtaining unit 12 generates a3-dimensional shape model based on actual measurement data from the3-dimensional measuring device 5, which is output to the coloringinformation estimating unit 13, as well as recording the 3-dimensionalshape model in the model data storage unit 14, as with the firstembodiment.

[0083] (Estimating Coloring Information)

[0084] The coloring information estimating unit 13 estimates thecoloring information which is the attributes information of the subject,based on the input information, and records reflection constants for thedisperse reflection and specular reflection making up the surfaceattributes in the model data storage unit 14. The following is a methodfor estimating the coloring information, with reference to the flowchartin FIG. 9.

[0085] First, in step S21, the flow starts. Subsequently, the followingprocessing is repeated for each coloring portion (estimation portion)for which the coloring information of the surface of the 3-dimensionalshape model is to be estimated.

[0086] In step S22, the correlation of the direction of the center ofthe light source at each coloring portion, the direction of observation,and the reflection intensity, is calculated and a correlation table iscompiled. This is achieved by calculating the direction of the center ofthe light source at each coloring portion, the direction of observation,and the position on the image, from the positional information of thephotography device 1, subject 3, and area light source 22 at the time ofphotography, and properties or settings information for the photographydevice 1 and area light source 22, and obtaining the reflectionintensity for each portion of the subject from the lightness on theimage.

[0087] In the event that the reflection intensity is unknown due to theplane being the back side on the image, behind other members, or outsideof the image, this is not included in the correlation table.

[0088] Next, in step S23, in order to express the correlation table asBRDF′ which is the apparent BRDF, approximation is made with theGaussian distribution type luminance-distribution-type reflection modelfunction shown in FIG. 7, for example, the correlation between thedirection of center of light source at each coloring portion, theobservation direction, and reflection intensity, in the correlationtable obtained in the preceding step, is represented with thereflectivity R_(j), G_(j), and B_(j), for each color band, and the anglevariables θ_(j), and ρ_(j) shown in FIG. 12.

[0089] The subscript j represents image Nos. 1, 2, and so on through m.Also, Cd_(R), Cd_(G), and Cd_(B) are diffuse reflection constants, withC_(S,σ) being a specular reflection constant. With an error vector as ε,this yields the determinant shown in FIG. 11. In the event that thereflectivity at the coloring portion is unknown, the corresponding lineis deleted from the determinant.

[0090] Next, in step S24, a set of constants optimal for approximatingthe correlation table with a reflection model function, i.e., the set(Cd_(R), Cd_(G), Cd_(B), Cs′, σ′) is decided. Various types ofmathematical approaches may be used for optimization; the presentinvention does not limit the approach used here.

[0091] As an example, in the event of using the least-square method, theperformance function is |ε|, the length of the error vector ε, and a setwherein the performance function is minimum is obtained. Though theperformance function handles all errors as being equivalent, aperformance function may be set which takes into consideration thereliability of each of the values.

[0092] Next, in step S25, the Gaussian distribution typeluminance-distribution-type reflection model function obtained in stepS24 is converted into a reflection model function relating to the lightsource, such as a point light source or a parallel (infinite-distance)light source wherein the incident angle to each portion of the subjectis constant.

[0093] The Gaussian distribution type reflection model function shown inFIG. 8 is used for the reflection model function. At this time, thedisperse reflection component is taken to be practically unaffected bythe expansion of the illumination light, and the aforementioned Cd_(R),Cd_(G), and Cd_(B) are recorded in the model data storage unit 14. Also,for the specular reflection component, as is obtained by the followingexpression which is a Gaussian distribution function synthesizing rule,and Cs and σ are recorded in the model data storage unit 14.

σ′² =σs ²+σ_(L) ²

σ² Cs=σ′ ² Cs′

[0094] Here, σ represents the standard deviation of the reflection modelfunction, σ′ represents the standard deviation of theluminance-distribution-type reflection model function, and σ_(L)represents the apparent Gauss radius of the luminance distribution ofthe illumination light.

[0095] The above-described processing is performed for all coloringportions.

[0096] (Reproduction by Computer Graphics)

[0097] A 3-dimensional image can be reproduced in the same way as withthe first embodiment, using the above-obtained 3-dimensional shape modeland the reflection model function which is the surface attributes ofeach portion on the 3-dimensional shape model.

[0098] While the present embodiment has been described using a Gaussiandistribution type reflection model function for the reflection modelfunction, there are cases wherein a Phong reflection model function ishandier at the time of reproduction by computer graphics. A Gaussianreflection model function and Phong reflection model function can becorrelated very closely by the conversion expression:

σ²=1/(n+5/6).

[0099] This may be used to record n instead of σ in step S26.

[0100] Also, while the present embodiment has been described withreference to an case wherein the image is made up of the three colors,R, G, and B, the present embodiment may use any arrangement as long ascorresponding reflection model functions can be described, such as onecolor, other multiple colors, spectrum information, polarizationinformation, etc.

[0101] Further, the coloring information estimating method described inthe present embodiment does not guarantee that all reflection constantsare uniquely obtained, but in such a case, techniques can be used suchas interpolation of coloring portions having unobtainable constantsbased on reflection constants at nearby coloring portions, substitutinga typical value, and so forth.

[0102] While the present invention has been described with reference towhat are presently considered to be the preferred embodiments, it is tobe understood that the invention is not limited to the disclosedembodiments. On the contrary, the invention is intended to cover variousmodifications and equivalent arrangements included within the spirit andscope of the appended claims. The scope of the following claims is to beaccorded the broadest interpretation so as to encompass all suchmodifications and equivalent structures and functions.

What is claimed is:
 1. A 3-dimensional image processing method, whichobtains 3-dimensional image data for reproducing an object as a3-dimensional image, comprising: a first step, for obtainingphotographed image data by illuminating an object with an area lightsource having the maximum angle of the solid angle for observing thelight source from each portion of said object is 5° or greater, andleaving a dark portion around said object; and a second step, forestimating surface attributes of said object from photographed imagedata obtained in said first step, and photography environmentinformation at the time of obtaining photographed image data in saidfirst step.
 2. A 3-dimensional image processing method according toclaim 1, wherein shape information of said object is also used forestimating the surface attributes of said object in said second step. 3.A 3-dimensional image processing method according to claim 1, whereinsaid photography environment information includes at least one ofinformation relating to spectral distribution of light from said arealight source, information relating to properties or settings of thephotography device to perform photography, information relating to theposition of said real object and said light source and said photographydevice, and information relating to the direction of photography withsaid photography device, in addition to information relating toluminance distribution of said area light source.
 4. A 3-dimensionalimage processing method according to claim 1, wherein said photographyimage data contains a plurality of sets of image data obtained bychanging at least one of the direction of illumination as to saidobject, and the direction of photography, when photographing.
 5. A3-dimensional image processing method according to claim 1, wherein thesurface attributes of each portion on the surface of said objectestimated in said second step are represented by a bi-directionalreflectance distribution function, which yields the reflectance forillumination light with a concentrated incident angle to said object. 6.A 3-dimensional image processing method according to claim 5, whereinsaid bi-directional reflectance distribution function is represented asa reflection model function, and wherein the reflection model functiontakes a plurality of reflection constants as function elements.
 7. A3-dimensional image processing method according to claim 6, wherein saidreflection model function is represented as a function with diffusereflection components and specular reflection components being linearlycombined; and wherein said reflection constants contain diffusereflection constants relating to diffuse reflection components andspecular reflection constants relating to specular reflectioncomponents.
 8. A 3-dimensional image processing method according toclaim 5, wherein, in said second step, said bi-directional reflectancedistribution function is estimated by deconvoluting of the change inluminance of said object obtained from said photography image data withthe luminance distribution of said area illumination light.
 9. A3-dimensional image processing method according to claim 8, wherein saidbi-directional reflectance distribution function is represented as areflection model function, and wherein the reflection model functiontakes a plurality of reflection constants as function elements.
 10. A3-dimensional image processing method according to claim 9, wherein saidreflection model function is represented as a function with diffusereflection components and specular reflection components being linearlycombined, and contains diffuse reflection constants relating to thediffuse reflection components and specular reflection constants relatingto the specular reflection components.
 11. A 3-dimensional imageprocessing method according to claim 1, wherein, in said second step, aluminance-distribution-type bi-directional reflectance distributionfunction, which indicates the distribution of reflectivity forillumination light having a predetermined distribution as to theincident angle for each portion of said object, is estimated, followingwhich said luminance-distribution-type bi-directional reflectancedistribution function is converted into a bi-directional reflectancedistribution function which yields the reflectance of each portion onthe surface of said object for the illumination light with aconcentrated incident angle to said object.
 12. A 3-dimensional imageprocessing method according to claim 11, wherein, in said second step,said luminance-distribution-type bi-directional reflectance distributionfunction is estimated based on said photographed image data and saidphotograph environment information.
 13. A 3-dimensional image processingmethod according to claim 12, wherein, in said second step, saidluminance-distribution-type bi-directional reflectance distributionfunction is converted into said bi-directional reflectance distributionfunction, based on said photography environment information.
 14. A3-dimensional image processing method according to claim 11, whereinsaid luminance-distribution-type reflection model function isrepresented as a function with diffuse reflection components andspecular reflection components being linearly combined, containingdiffuse reflection constants relating to the diffuse reflectioncomponents and specular reflection constants relating to the specularreflection components.
 15. A 3-dimensional image processing methodaccording to claim 11, wherein both the luminance distribution at thelight source portion of said area illumination light and the specularreflection components of said luminance-distribution-type bi-directionalreflectance distribution function can be described by Gaussiandistribution, and wherein the standard deviation σ of the specularreflection component of said bi-directional reflectance distributionfunction is obtained by the following relational expression:σ′²=σ²+σ_(L) ² wherein σ′ represents the standard deviation of thespecular reflection component of luminance-distribution-typebi-directional reflectance distribution function, and σ_(L) representsthe standard deviation of apparent luminance distribution at the lightsource portion of illumination light.
 16. A 3-dimensional imageprocessing device, which obtains 3-dimensional image data forreproducing an object as a 3-dimensional image, comprising: image dataobtaining means, for obtaining photographed image data by illuminatingan object with an area light source having the maximum angle of thesolid angle for observing the light source from each portion of saidobject is 5° or greater, and leaving a dark portion around said object;and surface attributes estimating means, for estimating surfaceattributes of said object from photographed image data obtained by saidimage data obtaining means, and photography environment information atthe time of obtaining photographed image data by said image dataobtaining means.
 17. A 3-dimensional image processing device accordingto claim 16, wherein shape information of said object is also used forestimating the surface attributes of said object by said surfaceattributes estimating means.
 18. A 3-dimensional image processing deviceaccording to claim 16, wherein said photography environment informationincludes at least one of information relating to spectral distributionof light from said area light source, information relating to propertiesor settings of the photography device to perform photography,information relating to the position of said real object and said lightsource and said photography device, and information relating to thedirection of photography with said photography device, in addition toinformation relating to luminance distribution of said area lightsource.
 19. A 3-dimensional image processing device according to claim16, wherein said photography image data contains a plurality of sets ofimage data obtained by changing at least one of the direction ofillumination as to said object, and the direction of photography, whenphotographing.
 20. A 3-dimensional image processing device according toclaim 16, wherein the surface attributes of each portion on the surfaceof said object estimated by said surface attributes estimating means arerepresented by a bi-directional reflectance distribution function, whichyields the reflectance for illumination light with a concentratedincident angle to said object.
 21. A 3-dimensional image processingdevice according to claim 20, wherein said bi-directional reflectancedistribution function is represented as a reflection model function, andwherein the reflection model function takes a plurality of reflectionconstants as function elements.
 22. A 3-dimensional image processingdevice according to claim 21, wherein said reflection model function isrepresented as a function with diffuse reflection components andspecular reflection components being linearly combined; and wherein saidreflection constants contain diffuse reflection constants relating todisperse reflection components and specular reflection constantsrelating to specular reflection components.
 23. A 3-dimensional imageprocessing device according to claim 20, wherein, by said surfaceattributes estimating means, said bi-directional reflectancedistribution function is estimated by deconvoluting of the change inluminance of said object obtained from said photography image data withthe luminance distribution of said area illumination light.
 24. A3-dimensional image processing device according to claim 23, whereinsaid bi-directional reflectance distribution function is represented asa reflection model function, and wherein the reflection model functiontakes a plurality of reflection constants as function elements.
 25. A3-dimensional image processing device according to claim 24, whereinsaid reflection model function is represented as a function with diffusereflection components and specular reflection components being linearlycombined, and contains diffuse reflection constants relating to thediffuse reflection components and specular reflection constants relatingto the specular reflection components.
 26. A 3-dimensional imageprocessing device according to claim 16, wherein, by said surfaceattributes estimating means, a luminance-distribution-typebi-directional reflectance distribution function, which indicates thedistribution of reflectivity for illumination light having apredetermined distribution as to the incident angle for each portion ofsaid object, is estimated, following which saidluminance-distribution-type bi-directional reflectance distributionfunction is converted into a bi-directional reflectance distributionfunction which yields the reflectance of each portion on the surface ofsaid object for the illumination light with a concentrated incidentangle to said object.
 27. A 3-dimensional image processing deviceaccording to claim 26, wherein, by said surface attributes estimatingmeans, said luminance-distribution-type bi-directional reflectancedistribution function is estimated based on said photographed image dataand said photograph environment information.
 28. A 3-dimensional imageprocessing device according to claim 27, wherein, by said surfaceattributes estimating means, said luminance-distribution-typebi-directional reflectance distribution function is converted into saidbi-directional reflectance distribution function, based on saidphotography environment information.
 29. A 3-dimensional imageprocessing device according to claim 26, wherein saidluminance-distribution-type reflection model function is represented asa function with diffuse reflection components and specular reflectioncomponents being linearly combined, containing diffuse reflectionconstants relating to the diffuse reflection components and specularreflection constants relating to the specular reflection components. 30.A 3-dimensional image processing device according to claim 26, whereinboth the luminance distribution at the light source portion of said areaillumination light and the specular reflection component of saidluminance-distribution-type bi-directional reflectance distributionfunction can be described by Gaussian distribution, and wherein thestandard deviation σ of the specular reflection component of saidbi-directional reflectance distribution function is obtained by thefollowing relational expression: σ′²=σ²+σ_(L) ² wherein σ′ representsthe standard deviation of the specular reflection component ofluminance-distribution-type bi-directional reflectance distributionfunction, and σ_(L) represents the standard deviation of apparentluminance distribution at the light source portion of illuminationlight.
 31. A 3-dimensional image processing system, comprising: a3-dimensional image processing device according to any of the claims 16through 30; a photography device for photographing said object andinputting image data to said 3-dimensional image processing device; andan area light source for irradiating illumination light on said object.32. A 3-dimensional image processing program, which obtains3-dimensional image data for reproducing an object as a 3-dimensionalimage, comprising: code for a first step, for obtaining photographedimage data by illuminating an object with an area light source havingthe maximum angle of the solid angle for observing the light source fromeach portion of said object is 5° or greater, and leaving a dark portionaround said object; and code for a second step, for estimating surfaceattributes of said object from photographed image data obtained in saidfirst step, and photography environment information at the time ofobtaining photographed image data in said first step.
 33. A3-dimensional image processing program according to claim 32, whereinshape information of said object is also used for estimating the surfaceattributes of said object in said second step.
 34. A 3-dimensional imageprocessing program according to claim 32, wherein said photographyenvironment information includes at least one of information relating tospectral distribution of light from said area light source, informationrelating to properties or settings of the photography device to performphotography, information relating to the position of said real objectand said light source and said photography device, and informationrelating to the direction of photography with said photography device,in addition to information relating to luminance distribution of saidarea light source.
 35. A 3-dimensional image processing programaccording to claim 32, wherein said photography image data contains aplurality of sets of image data obtained by changing at least one of thedirection of illumination as to said object, and the direction ofphotography, when photographing.
 36. A 3-dimensional image processingprogram according to claim 32, wherein the surface attributes of eachportion on the surface of said object estimated in said second step arerepresented by a bi-directional reflectance distribution function, whichyields the reflectance for illumination light with a concentratedincident angle to said object.
 37. A 3-dimensional image processingprogram according to claim 36, wherein said bi-directional reflectancedistribution function is represented as a reflection model function, andwherein the reflection model function takes a plurality of reflectionconstants as function elements.
 38. A 3-dimensional image processingprogram according to claim 37, wherein said reflection model function isrepresented as a function with diffuse reflection components andspecular reflection components being linearly combined; and wherein saidreflection constants contain diffuse reflection constants relating todisperse reflection components and specular reflection constantsrelating to specular reflection components.
 39. A 3-dimensional imageprocessing program according to claim 36, wherein, in said second step,said bi-directional reflectance distribution function is estimated bydeconvoluting of the change in luminance of said object obtained fromsaid photography image data with the luminance distribution of said areaillumination light.
 40. A 3-dimensional image processing programaccording to claim 39, wherein said bi-directional reflectancedistribution function is represented as a reflection model function, andwherein the reflection model function takes a plurality of reflectionconstants as function elements.
 41. A 3-dimensional image processingprogram according to claim 40, wherein said reflection model function isrepresented as a function with diffuse reflection components andspecular reflection components being linearly combined, and containsdiffuse reflection constants relating to the diffuse reflectioncomponents and specular reflection constants relating to the specularreflection components.
 42. A 3-dimensional image processing programaccording to claim 32, wherein, in said second step, aluminance-distribution-type bi-directional reflectance distributionfunction, which indicates the distribution of reflectivity forillumination light having a predetermined distribution as to theincident angle for each portion of said object, is estimated, followingwhich said luminance-distribution-type bi-directional reflectancedistribution function is converted into a bi-directional reflectancedistribution function which yields the reflectance of each portion onthe surface of said object for the illumination light with aconcentrated incident angle to said object.
 43. A 3-dimensional imageprocessing program according to claim 42, wherein, in said second step,said luminance-distribution-type bi-directional reflectance distributionfunction is estimated based on said photographed image data and saidphotograph environment information.
 44. A 3-dimensional image processingprogram according to claim 43, wherein, in said second step, saidluminance-distribution-type bi-directional reflectance distributionfunction is converted into said bi-directional reflectance distributionfunction, based on said photography environment information.
 45. A3-dimensional image processing program according to claim 42, whereinsaid luminance-distribution-type reflection model function isrepresented as a function with diffuse reflection components andspecular reflection components being linearly combined, containingdiffuse reflection constants relating to the diffuse reflectioncomponents and specular reflection constants relating to the specularreflection components.
 46. A 3-dimensional image processing programaccording to claim 42, wherein both the luminance distribution at thelight source portion of said area illumination light and the specularreflection component of said luminance-distribution-type bi-directionalreflectance distribution function can be described by Gaussiandistribution, and wherein the standard deviation σ of the specularreflection component of said bi-directional reflectance distributionfunction is obtained by the following relational expression:σ′²=σ²+σ_(L) ² wherein σ′ represents the standard deviation of thespecular reflection component of luminance-distribution-typebi-directional reflectance distribution function, and σ_(L) representsthe standard deviation of apparent luminance distribution at the lightsource portion of illumination light.